feat: Added new API method `withLlmCustomAttributes` to run a function in a LLM context #2437

MikeVaz · 2024-08-02T15:55:59Z

Hi team. Here is our AIM's humble effort to implement adding LLM custom attributes

Description

During GA for the AIM product, custom attributes on the transaction level are prefixed with llm. were set up such that they would be added to all LLM events generated in that transaction from that point onwards, however, there has been demand for a more granular version of this.

This would be intended to support use cases such as A/B testing individual parts of flows (for example, the first call out to the LLM in a transaction might go to flow A, and the resulting second call in the transaction might go down flow B based on output from a tool).

Solution highlight

We went with an option 4 based on the DACI implementing withLlmCustomAttributes() method
In addition setLlmCustomAttributesCallback can be provided but removed from this PR for now.
We originally tried to use an existing context manager on the agent instance to run a LLM context. this.agent._contextManager.runInContext.
But the issue below made us move to using an isolated context manager on the transaction. transaction._llmContextManager = new AsyncLocalStorage();.
Provided attributes will be normalized and later extracted from the context to be added to the custom event
A new method withLlmCustomAttributes added to the APIs plus recordEvent method updated for openai, bedrock and langchain instrumentations.

How to Test

We have tested it in our sample application. We have a dedicated branch.
Changes are covered by unit and integration tests as well.

Related Issues

One issue is revealed when Langchain versioned test is failing. We would like to consult with the team on that. I suspect something in the langchain flow overrides the context viathis.agent._contextManager.setContext({....}). A solutions could be to go with a dedicated isolated context manager transaction._llmContextManager = new AsyncLocalStorage();.

Problem

Consider we have a flow like below

└  Start Transaction
     └ withLlmCustomAttributes {llm.attribute: 1}
         │- Open AI call #1
         │   └ recordEvent with llm context {llm.attribute:1}
         └ switch context via agent._contextManager.runInContext({custom: 1})
                └ Open AI call #2
                     └ recordEvent when llm context lost

Solution 1 (currently implemented)

We created an isolated context manager so it would never collide with agent's one.

Solution 2

Alternatively we can refactor agent's context manager to store parent context in a WeakMap

    setParentContext(child, parent) {
        this._parentContexMap = this._parentContextMap || new WeaMap()
        this._parentContextMap.set(child, parent)
    }
    runInContext(context, callback, cbThis, args) {
         const toInvoke = cbThis ? callback.bind(cbThis) : callback
         
         // Something like
         this.setParentContext(context, this.getContext())
         
         if (args) {
           return this._asyncLocalStorage.run(context, toInvoke, ...args)
         }
     
         return this._asyncLocalStorage.run(context, toInvoke)
       }

Now we can traverse related contexts to aggregate all llm context attributes attached along the way.

CLAassistant · 2024-08-02T15:56:06Z

All committers have signed the CLA.

CLAassistant · 2024-08-02T15:56:06Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 3 committers have signed the CLA.

❌ mvazhenin
❌ MikeVaz
❌ RyanKadri

mvazhenin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

test/versioned/langchain/runnables.tap.js

api.js

lib/instrumentation/aws-sdk/v3/bedrock.js

lib/instrumentation/langchain/common.js

lib/instrumentation/openai.js

package.json

test/versioned/langchain/runnables.tap.js

api.js

bizob2828 · 2024-08-02T21:17:30Z

api.js

@@ -1902,4 +1902,47 @@ API.prototype.ignoreApdex = function ignoreApdex() {
  transaction.ignoreApdex = true
 }

+/**
+ * Runs a function synchronously within a provided LLM custom attributes context and returns its return value.


I don't think it has to be synchronously. The context manager should properly bind all functions run within the context

I went with how they describe it in the nodejs docs.

Runs a function synchronously within a context and returns its return value. The store is not accessible outside of the callback function. The store is accessible to any asynchronous operations created within the callback.

The store is accessible to any asynchronous operations created within the callback.

MikeVaz · 2024-08-05T22:05:47Z

@bizob2828 @jsumners-nr I have updated my PR to address most of the feedback. I also got an idea why langchain was failing. Basically it would record events but only when virtual_llm = true and is_response = true events has the same context. The rest would somehow lose the context. Do you have ideas?

jsumners-nr

With the latest changes, I am not seeing any test failures. What does a test look like that generates the issue you are seeing?

test/versioned/aws-sdk-v3/bedrock-chat-completions.tap.js

test/versioned/langchain/runnables.tap.js

MikeVaz · 2024-08-06T15:19:45Z

With the latest changes, I am not seeing any test failures. What does a test look like that generates the issue you are seeing?

It is not failing since I am filtering events

      const responses = events.filter((event) => {
          const [, chainEvent] = event
          return chainEvent.virtual_llm === true && chainEvent.is_response === true
        })

Removing filter would make test to fail

lib/util/llm-utils.js

test/unit/util/llm-utils.test.js

api.js

test/versioned/openai/chat-completions.tap.js

codecov · 2024-08-20T18:47:06Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.15%. Comparing base (0c2ee2f) to head (21e4b5f).
Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2437      +/-   ##
==========================================
+ Coverage   96.20%   97.15%   +0.94%     
==========================================
  Files         288      289       +1     
  Lines       45305    45415     +110     
==========================================
+ Hits        43586    44123     +537     
+ Misses       1719     1292     -427

Flag	Coverage Δ
integration-tests-cjs-18.x	`73.80% <39.17%> (?)`
integration-tests-cjs-20.x	`73.82% <39.17%> (?)`
integration-tests-cjs-22.x	`73.85% <39.17%> (?)`
integration-tests-esm-18.x	`49.15% <39.17%> (?)`
integration-tests-esm-20.x	`49.15% <39.17%> (?)`
integration-tests-esm-22.x	`49.18% <39.17%> (?)`
unit-tests-18.x	`88.67% <82.69%> (+<0.01%)`	⬆️
unit-tests-20.x	`88.67% <82.69%> (+<0.01%)`	⬆️
unit-tests-22.x	`88.68% <82.69%> (+<0.01%)`	⬆️
versioned-tests-18.x	`78.84% <88.46%> (-0.06%)`	⬇️
versioned-tests-20.x	`78.88% <88.46%> (-0.03%)`	⬇️
versioned-tests-22.x	`78.88% <88.46%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

test/unit/api/api-llm.test.js

bizob2828

feedback from the last round

lib/util/llm-utils.js

lib/instrumentation/aws-sdk/v3/bedrock.js

lib/instrumentation/langchain/common.js

lib/instrumentation/openai.js

Co-authored-by: Bob Evans <robert.evans25@gmail.com>

…ode-newrelic into setLlmCustomAttributes

test/unit/api/api-llm.test.js

* Bump fast-xml-parser and @aws-sdk/client-lambda Bumps [fast-xml-parser](https://github.com/NaturalIntelligence/fast-xml-parser) and [@aws-sdk/client-lambda](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-lambda). These dependencies needed to be updated together. Updates `fast-xml-parser` from 4.2.4 to 4.2.5 - [Release notes](https://github.com/NaturalIntelligence/fast-xml-parser/releases) - [Changelog](https://github.com/NaturalIntelligence/fast-xml-parser/blob/master/CHANGELOG.md) - [Commits](NaturalIntelligence/fast-xml-parser@v4.2.4...v4.2.5) Updates `@aws-sdk/client-lambda` from 3.358.0 to 3.359.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-lambda/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.359.0/clients/client-lambda) --- updated-dependencies: - dependency-name: fast-xml-parser dependency-type: indirect - dependency-name: "@aws-sdk/client-lambda" dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump protobufjs from 7.2.3 to 7.2.4 Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.3 to 7.2.4. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-v7.2.3...protobufjs-v7.2.4) --- updated-dependencies: - dependency-name: protobufjs dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore: added node 20 and drop node 14 in CI * chore: fixed deps with CVEs * test: skip Next.js 13.4.13 until we can fix the instrumentation * fix: updated instrumentation to skip registering middleware instrumentation as it runs in a worker thread now and our agent cannot properly track async context * chore: removes skipping of tests on 13.4.13 and above * chore: change node engine to 16 * Setting version to v0.6.0. * Adds auto-generated release notes. * chore: Edited CHANGELOG.md Signed-off-by: mrickard <maurice@mauricerickard.com> * test: update versioned test helper to handle next@13.4.15 changes * chore: update path for ritm * remove slack link as it is decommissioned * chore: updated peer dep to the unreleased version of agent that this instrumentation will now require * chore: updated agent to latest * Setting version to v0.7.0. * Adds auto-generated release notes. * chore: changelog edits * chore: updated @newrelic/test-utilities to latest * chore(deps): bump @babel/traverse Bumps and [@babel/traverse](https://github.com/babel/babel/tree/HEAD/packages/babel-traverse). These dependencies needed to be updated together. Updates `@babel/traverse` from 7.17.3 to 7.23.2 - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.23.2/packages/babel-traverse) Updates `@babel/traverse` from 7.20.0 to 7.23.2 - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.23.2/packages/babel-traverse) --- updated-dependencies: - dependency-name: "@babel/traverse" dependency-type: indirect - dependency-name: "@babel/traverse" dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * test: skip running Next 14+ versioned tests on Node 16 as support was dropped * fix: package.json & package-lock.json to reduce vulnerabilities The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-AXIOS-6032459 * chore(deps-dev): bump follow-redirects from 1.15.3 to 1.15.4 Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.3 to 1.15.4. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.15.3...v1.15.4) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * test: updated test assertions based on segment tree changes in 14.1.0 of Next.js * test: updated test assertions based on segment tree changes in 14.1.0 of Next.js * refactor: Updated instrumentation to construct spec objects at instrumentation * Setting version to v0.8.0. * Adds auto-generated release notes. * Update CHANGELOG.md * feat: Added a shim to externalize all 3rd party libraries the Node.js agent instruments * feat: Added a test suite for App Router. * chore(deps-dev): bump follow-redirects from 1.15.5 to 1.15.6 Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.5 to 1.15.6. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.15.5...v1.15.6) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Updated CI process for releases (newrelic#183) * chore: release v0.9.0 (newrelic#184) Co-authored-by: jsumners-nr <gh-actions-jsumners-nr@github.com> Co-authored-by: James Sumners <jsumners@newrelic.com> * ci: removed changelog.json file (newrelic#185) * ci: Removed `use_new_release` input from prepare release workflow (newrelic#186) * test: Added targets for compatibility reporting (newrelic#187) * chore: Enabled quiet mode for CI runs (newrelic#188) * docs: Updated targets to include minimum agent version for compatibility repo (newrelic#189) * docs: Added FAQs to assist with common issues with next.js instrumentation (newrelic#190) * chore: Made pre-commit hook require dependency changes (newrelic#191) * docs: updated FAQs and README with app router examples (newrelic#192) * fix: add missing quotation mark in faq docs (newrelic#202) * chore(deps-dev): bump @grpc/grpc-js from 1.9.9 to 1.10.9 (newrelic#203) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump braces from 3.0.2 to 3.0.3 (newrelic#204) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * security(deps): bump ws (newrelic#206) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Added Node 22 to CI (newrelic#193) * chore: release v0.10.0 (newrelic#210) * chore: Fixed copy paste error in post release workflow (newrelic#2329) * fix: Pinned dependenices of node-gyp that dropped support for Node 16 in patch releases (newrelic#2333) * fix: Refactored benchmark tests to complete async functions (newrelic#2334) Signed-off-by: mrickard <maurice@mauricerickard.com> * chore: Revert "fix: Pinned dependenices of node-gyp that dropped support for Node 16 in patch releases (newrelic#2333)" (newrelic#2335) * feat: Added support for account level governance of AI Monitoring (newrelic#2326) * test: Fixed recordMiddlewawre benchmark test (newrelic#2338) * chore: release v11.23.0 (newrelic#2340) Co-authored-by: jsumners-nr <gh-actions-jsumners-nr@github.com> Co-authored-by: James Sumners <jsumners@newrelic.com> * docs: Updated compatibility report (newrelic#2342) Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> * fix: Updated redis v4 instrumentation to work with transactions(multi/exec) (newrelic#2343) * chore: release v11.23.1 (newrelic#2344) * docs: Updated compatibility report (newrelic#2345) Co-authored-by: bizob2828 <1874937+bizob2828@users.noreply.github.com> * chore: Always upload status logs in compat report CI (newrelic#2341) * ci: Updated `bin/create-docs-pr` to create an empty array if changelog.json is missing security (newrelic#2348) * ci: increase the limit of installs from 2 to a bigger number (newrelic#2346) * ci: Changed the default project idea for our org board (newrelic#2353) * ci: Changed the default project idea for our org board (newrelic#2355) * ci: Updated board workflow to use new graphql calls to add items to project board (newrelic#2357) * ci: Fixed issue with obtaining node id for issues in add-to-board (newrelic#2360) * ci: Fixed syntax issue with parsing jq (newrelic#2362) * test: Updated benchmark test results to output result files (newrelic#2350) Signed-off-by: mrickard <maurice@mauricerickard.com> Co-authored-by: Bob Evans <robert.evans25@gmail.com> * docs: Removed out of date ROADMAP_Node.md from root of project (newrelic#2367) * refactor: consolidated adding issue/pr to board and assigning the appropriate status into 1 step (newrelic#2368) * refactor: fixed syntax error with add to board workflow (newrelic#2370) * chore: fix board refactor (newrelic#2371) * ci: Added benchmark test GitHub Action (newrelic#2366) Signed-off-by: mrickard <maurice@mauricerickard.com> * feat: Added support for fs.glob in Node 22+ (newrelic#2369) * test: Removed server.start in grpc tests as it is deprecated and no longer needed (newrelic#2372) * fix: Updated cassandra-driver instrumentation to properly trace promise based executions (newrelic#2351) * ci: Include date created when adding new issue/pr to board (newrelic#2374) * ci: Pin Node 22 to 22.4.1 (newrelic#2375) * fix: Updated aws-sdk v3 instrumentation to custom middleware last to properly get the external http span to add aws.* attributes (newrelic#2382) * chore: Reverted "ci: Pin Node 22 to 22.4.1" (newrelic#2383) * refactor: remove examples/api/ (newrelic#2381) * chore: release v11.23.2 (newrelic#2391) * docs: Updated compatibility report (newrelic#2392) * chore: Updated dashboard links in developer-setup.md (newrelic#2397) * refactor: Removed `Supportability/Features/ESM/UnsupportedLoader` as it is no longer applicable in Node.js 18+ (newrelic#2393) * feat!: Dropped support for Node.js 16 (newrelic#2394) * feat!: Updated `mongodb` instrumentation to drop support for versions 2 and 3 (newrelic#2398) * test: Updated minimum version of lesser used versions of 3rd party li… (newrelic#2399) * chore: Verified MySQL host:port metric is recorded (newrelic#2400) * feat!: Removed instrumentation for `director` (newrelic#2402) * chore: Add test configs for defined targets in the aws test suite (newrelic#2403) * feat!: Removed legacy context manager (newrelic#2404) * docs: Updated compatibility report (newrelic#2401) Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> * feat!: Removed support for `redis` < 2.6.0 (newrelic#2405) * chore: Switch to using Node built-in test runner (newrelic#2387) * feat: Added `server.address` to amqplib spans (newrelic#2406) * refactor: Moved relevant nextjs instrumentation and rely on agent commons * chore: Added producer and consumer metrics to kafkajs instrumentation (newrelic#2407) * chore: Updated `@newrelic/native-metrics` to 11.0.0 * test: Removed mongodb-esm tests as they are not atomic and conflicting with mongodb tests in CI * chore: release v12.0.0 (newrelic#2418) * docs: Updated compatibility report (newrelic#2415) * docs: Updated examples to properly use specs (newrelic#2422) * fix: Pick log message from merging object in Pino instrumentation (newrelic#2421) * test: Updated custom test reporter to only log failed tests when there are failures (newrelic#2425) * fix: typo in doc header (newrelic#2433) * chore: Converted agent unit tests to node:test (newrelic#2414) * test: Restored mongodb-esm tests (newrelic#2434) * docs: Updated compatibility report (newrelic#2435) Co-authored-by: bizob2828 <1874937+bizob2828@users.noreply.github.com> * chore: Added entity relationship attributes to SQS segments (newrelic#2436) * test: Moved pkgVersion to collection-common to avoid a conflict with ESM tests (newrelic#2438) * chore: Limited superagent tests to avoid new breaking release (newrelic#2439) * chore: Fixed mongodb-esm tests in combination with security agent (newrelic#2444) * docs: Updated compatibility report (newrelic#2440) Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> * chore: Remove promise resolvers from callback based agent unit tests (newrelic#2450) * chore: Added TLS verification for Redis (newrelic#2446) * test: Updated tls redis tests to work with older versions of redis v4 (newrelic#2454) * chore: release v12.1.0 (newrelic#2455) Co-authored-by: svetlanabrennan <gh-actions-svetlanabrennan@github.com> Co-authored-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com> Co-authored-by: Maurice Rickard <maurice@mauricerickard.com> * docs: Updated compatibility report (newrelic#2452) Co-authored-by: svetlanabrennan <50715937+svetlanabrennan@users.noreply.github.com> * fix: Updated the `kafkajs` node metrics to remove `/Named` from the name (newrelic#2458) * chore: Removed limit on superagent versioned testing (newrelic#2456) * docs: Updated compatibility report (newrelic#2460) Co-authored-by: bizob2828 <1874937+bizob2828@users.noreply.github.com> * refactor: Updated pino instrumentation to separate the wrapping of asJson into its own function (newrelic#2464) * fix: Updated redis instrumentation to parse host/port when a url is not provided (newrelic#2463) * fix: Updated amqplib instrumentation to properly parse host/port from connect (newrelic#2461) * chore: release v12.1.1 (newrelic#2472) Co-authored-by: svetlanabrennan <gh-actions-svetlanabrennan@github.com> Co-authored-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com> Co-authored-by: Bob Evans <revans@newrelic.com> * docs: Updated compatibility report (newrelic#2474) Co-authored-by: svetlanabrennan <50715937+svetlanabrennan@users.noreply.github.com> * test: Skip `@koa/router@13.0.0` because of failures (newrelic#2478) * docs: Updated compatibility report (newrelic#2480) Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> * feat: Added instrumentation support for Express 5 beta (newrelic#2476) This will be experimental until express@5.0.0 is generally available * docs: Remove reference to @newrelic/next in README (newrelic#2479) * docs: Updated compatibility report (newrelic#2483) Co-authored-by: bizob2828 <1874937+bizob2828@users.noreply.github.com> * chore: Reverted to upstream require-in-the-middle (newrelic#2473) * chore: Updated aggregators unit tests to node:test (newrelic#2481) * fix: Updated koa instrumentation to properly get the matched route name and to handle changes in `@koa/router@13.0.0` (newrelic#2486) * docs: Updated compatibility report (newrelic#2487) Co-authored-by: bizob2828 <1874937+bizob2828@users.noreply.github.com> * chore: release v12.2.0 (newrelic#2492) Co-authored-by: svetlanabrennan <gh-actions-svetlanabrennan@github.com> Co-authored-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com> * ci: Updated codecov action sha to post coverage from forks. Added flag to fail ci if it fails to upload report (newrelic#2490) * chore: Updated test-utils dependency and added matrix-count only (newrelic#2494) * chore: Remove examples/shim (newrelic#2484) * chore: Fixed linting scripts (newrelic#2497) * fix: Improved AWS Lambda event detection (newrelic#2498) * docs: Updated compatibility report (newrelic#2493) Co-authored-by: svetlanabrennan <50715937+svetlanabrennan@users.noreply.github.com> * feat: Added new API method `withLlmCustomAttributes` to run a function in a LLM context (newrelic#2437) The context will be used to assign custom attributes to every LLM event produced within the function * chore: Converted context-manager unit tests to node:test (newrelic#2508) Co-authored-by: Bob Evans <robert.evans25@gmail.com> * test: Converted the api unit tests to `node:test` (newrelic#2516) * chore: release v12.3.0 (newrelic#2522) * docs: cleaned up formatting of api.js to properly inject example snippets when rendering on API docs site (newrelic#2524) * docs: Updated compatibility report (newrelic#2523) Co-authored-by: bizob2828 <1874937+bizob2828@users.noreply.github.com> * test: Convert db unit tests to node:test (newrelic#2514) * chore: Convert `config` to `node:test` (newrelic#2517) * test: Replace distributed tracing tests with `node:test` (newrelic#2527) * test: Convert grpc, lib, and utilization tests to `node:test` (newrelic#2532) * docs: Updated Next.js Otel cloud provider FAQ (newrelic#2537) * docs: Updated formatting of cloud-providers.md (newrelic#2538) * chore: Added a match function for tests (newrelic#2541) * fix: Fixed detection of REST API type payloads in AWS Lambda (newrelic#2543) * chore: release v12.3.1 (newrelic#2544) * docs: Updated compatibility report (newrelic#2545) * test: Migrated tests in `test/unit/instrumentation` to use `node:test` (newrelic#2531) * chore: Converted collector unit tests to node:test (newrelic#2510) * test: Converted `llm-events` tests to use `node:test` (newrelic#2535) * chore: Added CI for publishing agent as Azure site extension (newrelic#2488) Signed-off-by: mrickard <maurice@mauricerickard.com> Co-authored-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com> Co-authored-by: James Sumners <jsumners@newrelic.com> * chore: Converted errors unit tests to node:test (newrelic#2540) Co-authored-by: Bob Evans <robert.evans25@gmail.com> * feat: Added Azure site extension installation scripts (newrelic#2448) Co-authored-by: James Sumners <jsumners@newrelic.com> * test: Migrated `test/unit/util` to use `node:test` (newrelic#2546) * chore: Disable express@5 in versioned tests (newrelic#2553) * docs: Updated compatibility report (newrelic#2554) Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> * feat: Added support for `express@5` (newrelic#2555) * test: Migrated `test/unit/spans` to use `node:test` (newrelic#2556) * feat: Provided ability to disable instrumentation for a 3rd party package (newrelic#2551) * fix: Nuget pack generates packagName.semver and not packageName-semver (newrelic#2557) Signed-off-by: mrickard <maurice@mauricerickard.com> * chore: Document emitted events (newrelic#2561) * chore: release v12.4.0 (newrelic#2560) Co-authored-by: jsumners-nr <gh-actions-jsumners-nr@github.com> Co-authored-by: James Sumners <jsumners@newrelic.com> Co-authored-by: Bob Evans <revans@newrelic.com> * docs: Updated compatibility report (newrelic#2562) Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> * test: Convert `metric` and `metrics-recorder` tests to `node:test` (newrelic#2552) * fix: Ensured README displays for Azure site extension (newrelic#2564) Signed-off-by: mrickard <maurice@mauricerickard.com> * chore: Updated serverless unit tests to node:test (newrelic#2549) * feat: Added utilization info for ECS (newrelic#2565) * chore: release v12.5.0 (newrelic#2567) Co-authored-by: jsumners-nr <gh-actions-jsumners-nr@github.com> Co-authored-by: James Sumners <jsumners@newrelic.com> * docs: Updated compatibility report (newrelic#2568) Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> * chore: Reduce koa-router version to enable CI (newrelic#2573) * docs: Updated compatibility report (newrelic#2574) Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> * chore: Migrate block of unit tests to `node:test` (newrelic#2570) * chore: Migrate second block of unit tests to `node:test` (newrelic#2572) * ci: Added workflow run trigger to Azure site extension publish job (newrelic#2575) Signed-off-by: mrickard <maurice@mauricerickard.com> * test: Removed transitive deps from versioned tests as they will auto-install if required as peer deps (newrelic#2580) * test: Updated koa-router to tests to handle bug fixes from 13.0.1 (newrelic#2578) * test: Updated a missing `minSupported` in aws-sdk-v3 versioned tests (newrelic#2582) * docs: Updated compatibility report (newrelic#2581) * chore: Removed noisy test log (newrelic#2583) * test: Updated fastify versioned tests to work with `fastify@5.0.0` (newrelic#2584) * test: Fixed @koa/router tests. path-to-regex differs between @koa/router and koa-router now (newrelic#2587) * test: Updated how we handle the koa-router nuance of wildcard routes (newrelic#2588) * docs: Updated compatibility report (newrelic#2589) * test: Convert transaction* and urltils tests to `node:test` (newrelic#2585) * chore(deps): Udpated @newrelic/security-agent to v2.0.0 (newrelic#2594) * fix: Fixed handling of Pino merging object (newrelic#2600) * chore: release v12.5.1 (newrelic#2602) * chore: Migrate block of unit tests to node:test (newrelic#2593) * docs: Updated compatibility report (newrelic#2601) Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> * test: Migrated `test/unit/shim` to `node:test` (newrelic#2599) * test: Migrated `test/versioned/express` to `node:test` (newrelic#2609) * chore: Migrate block of unit tests to `node:test` (newrelic#2607) * chore: Migrate block of unit tests to node:test (newrelic#2604) * test: Updated tests that relied on `tspl` by awating the `plan.completed` instead of calling `end` to avoid flaky tests (newrelic#2610) * test: Migrated `test/versioned/amqplib` to `node:test` (newrelic#2612) * test: Updated the mininum version of pg-native in pg-esm tests to align with the pg tests (newrelic#2616) * chore: Upgraded `import-in-the-middle` to work around a bug introduced in 1.11.1 (newrelic#2618) * docs: Updated compatibility report (newrelic#2614) * test: Migrated `aws-sdk-v2` and `aws-sdk-v3` tests to `node:test` (newrelic#2620) * test: Migrated last group of unit tests to `node:test` (newrelic#2624) * test: Migrated unit tests to `node:test` (newrelic#2623) * docs: Remove SECURITY.md (newrelic#2633) * docs: Updated match custom-assertion jsdoc (newrelic#2636) * test: Migrated bluebird versioned tests to `node:test` (newrelic#2635) * docs: Updated compatibility report (newrelic#2637) * chore: Migrate `fastify` tests to `node:test` (newrelic#2632) Co-authored-by: Bob Evans <robert.evans25@gmail.com> * docs: Updated compatibility report (newrelic#2638) * chore: Migrate `bunyan`, `pino`, and `winston` tests to `node:test` (newrelic#2634) Co-authored-by: Bob Evans <robert.evans25@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: mrickard <maurice@mauricerickard.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Bob Evans <revans@newrelic.com> Co-authored-by: Bob Evans <robert.evans25@gmail.com> Co-authored-by: Mikko Kotamies <mkotamies@gmail.com> Co-authored-by: mrickard <gh-actions-mrickard@github.com> Co-authored-by: mrickard <maurice@mauricerickard.com> Co-authored-by: Jessica Lopatta <jlopatta@newrelic.com> Co-authored-by: Naresh Nishad <nc.nishad10@gmail.com> Co-authored-by: bizob2828 <gh-actions-bizob2828@github.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: svetlanabrennan <gh-actions-svetlanabrennan@github.com> Co-authored-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com> Co-authored-by: James Sumners <jsumners@newrelic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: jsumners-nr <gh-actions-jsumners-nr@github.com> Co-authored-by: Alisson Leal <alissonsleal@hotmail.com> Co-authored-by: Node Agent Bot <97628601+newrelic-node-agent-team@users.noreply.github.com> Co-authored-by: jsumners-nr <150050532+jsumners-nr@users.noreply.github.com> Co-authored-by: bizob2828 <1874937+bizob2828@users.noreply.github.com> Co-authored-by: Amy Chisholm <achisholm@newrelic.com> Co-authored-by: kmudduluru <kmudduluru@newrelic.com> Co-authored-by: Brian Hensley <48165493+brnhensley@users.noreply.github.com> Co-authored-by: Jamie Penney <jamiepenney@users.noreply.github.com> Co-authored-by: Webrealizer <webrealizer@gmail.com> Co-authored-by: Sumit Suthar <ssuthar@newrelic.com> Co-authored-by: Vaughn Woerpel <vaughnwoerpel@gmail.com>

MikeVaz and others added 13 commits July 16, 2024 12:12

feat: Set LLM events custom attributes

2361240

fix: Debug statements

4dd3be8

fix: Example

55da778

fix: Add guards

720be07

fix: Add unit test

878d6b5

feat: WithLlmCustomAttributes

de8b498

feat: Merge parent context into children

dc0261a

Merge branch 'newrelic:main' into setLlmCustomAttributes

07cbcab

feat: Option 4

08927ec

fix: Unnecessary test

8b0cd84

fix: Test name

7ab7bcd

fix: Integration tests

0b2f2d6

fix: Remove extra npm scripts

f75dabf

MikeVaz commented Aug 2, 2024

View reviewed changes

test/versioned/langchain/runnables.tap.js Outdated Show resolved Hide resolved

jsumners-nr reviewed Aug 2, 2024

View reviewed changes

MikeVaz added 3 commits August 2, 2024 09:13

fix: Remove extra npm scripts

19a9e80

fix: PR feedback

24a13bd

fix: PR feedback

af4aafa

jsumners-nr reviewed Aug 2, 2024

View reviewed changes

test/versioned/langchain/runnables.tap.js Outdated Show resolved Hide resolved

bizob2828 changed the title ~~feat: withLlmCustomAttributes~~ feat: Added new API method withLlmCustomAttributes Aug 2, 2024

bizob2828 reviewed Aug 2, 2024

View reviewed changes

api.js Outdated Show resolved Hide resolved

bizob2828 reviewed Aug 2, 2024

View reviewed changes

MikeVaz added 2 commits August 5, 2024 11:48

fix: PR feedback

d428aaa

fix: Unit test and pr feedback

d2264ad

MikeVaz marked this pull request as ready for review August 5, 2024 22:05

jsumners-nr reviewed Aug 6, 2024

View reviewed changes

test/versioned/aws-sdk-v3/bedrock-chat-completions.tap.js Show resolved Hide resolved

test/versioned/langchain/runnables.tap.js Outdated Show resolved Hide resolved

Merge branch 'newrelic:main' into setLlmCustomAttributes

ba516b0

bizob2828 requested changes Aug 19, 2024

View reviewed changes

lib/util/llm-utils.js Outdated Show resolved Hide resolved

test/unit/util/llm-utils.test.js Show resolved Hide resolved

api.js Outdated Show resolved Hide resolved

test/versioned/openai/chat-completions.tap.js Outdated Show resolved Hide resolved

bizob2828 mentioned this pull request Aug 19, 2024

Update context manager to store more than active segment #2495

Open

MikeVaz and others added 2 commits August 20, 2024 10:44

fix: PR feedback

6e629f9

Merge branch 'newrelic:main' into setLlmCustomAttributes

a8cbb8a

MikeVaz requested a review from bizob2828 August 20, 2024 17:48

fix: PR feedback

6d3bd2d

bizob2828 reviewed Aug 21, 2024

View reviewed changes

test/unit/api/api-llm.test.js Outdated Show resolved Hide resolved

bizob2828 requested changes Aug 21, 2024

View reviewed changes

MikeVaz and others added 5 commits August 21, 2024 10:02

Update lib/instrumentation/openai.js

1f2d240

Co-authored-by: Bob Evans <robert.evans25@gmail.com>

Update lib/instrumentation/langchain/common.js

8cc326f

Co-authored-by: Bob Evans <robert.evans25@gmail.com>

Update lib/instrumentation/aws-sdk/v3/bedrock.js

cab2f74

Co-authored-by: Bob Evans <robert.evans25@gmail.com>

fix: Improve test coverage

67a35de

Merge branch 'setLlmCustomAttributes' of https://github.com/MikeVaz/n…

d8353ff

…ode-newrelic into setLlmCustomAttributes

jsumners-nr previously approved these changes Aug 21, 2024

View reviewed changes

fix: More unit test and PR feedback

cb423d6

MikeVaz dismissed jsumners-nr’s stale review via cb423d6 August 21, 2024 19:04

MikeVaz requested a review from bizob2828 August 21, 2024 19:04

MikeVaz commented Aug 21, 2024

View reviewed changes

test/unit/api/api-llm.test.js Show resolved Hide resolved

chore: Addressed code review feedback

21e4b5f

bizob2828 approved these changes Aug 22, 2024

View reviewed changes

bizob2828 changed the title ~~feat: Added new API method withLlmCustomAttributes~~ feat: Added new API method withLlmCustomAttributes to run a function in a LLM context Aug 22, 2024

bizob2828 merged commit 57e6be9 into newrelic:main Aug 22, 2024
28 checks passed

github-actions bot mentioned this pull request Aug 27, 2024

chore: release v12.3.0 #2522

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Added new API method `withLlmCustomAttributes` to run a function in a LLM context #2437

feat: Added new API method `withLlmCustomAttributes` to run a function in a LLM context #2437

MikeVaz commented Aug 2, 2024 •

edited

Loading

CLAassistant commented Aug 2, 2024 •

edited

Loading

CLAassistant commented Aug 2, 2024

bizob2828 Aug 2, 2024

MikeVaz Aug 5, 2024

bizob2828 Aug 13, 2024

MikeVaz commented Aug 5, 2024 •

edited

Loading

jsumners-nr left a comment

MikeVaz commented Aug 6, 2024

codecov bot commented Aug 20, 2024 •

edited

Loading

bizob2828 left a comment

feat: Added new API method withLlmCustomAttributes to run a function in a LLM context #2437

feat: Added new API method withLlmCustomAttributes to run a function in a LLM context #2437

Conversation

MikeVaz commented Aug 2, 2024 • edited Loading

Description

Solution highlight

How to Test

Related Issues

Problem

Solution 1 (currently implemented)

Solution 2

CLAassistant commented Aug 2, 2024 • edited Loading

CLAassistant commented Aug 2, 2024

bizob2828 Aug 2, 2024

Choose a reason for hiding this comment

MikeVaz Aug 5, 2024

Choose a reason for hiding this comment

bizob2828 Aug 13, 2024

Choose a reason for hiding this comment

MikeVaz commented Aug 5, 2024 • edited Loading

jsumners-nr left a comment

Choose a reason for hiding this comment

MikeVaz commented Aug 6, 2024

codecov bot commented Aug 20, 2024 • edited Loading

Codecov Report

bizob2828 left a comment

Choose a reason for hiding this comment

feat: Added new API method `withLlmCustomAttributes` to run a function in a LLM context #2437

feat: Added new API method `withLlmCustomAttributes` to run a function in a LLM context #2437

MikeVaz commented Aug 2, 2024 •

edited

Loading

CLAassistant commented Aug 2, 2024 •

edited

Loading

MikeVaz commented Aug 5, 2024 •

edited

Loading

codecov bot commented Aug 20, 2024 •

edited

Loading